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Chris  Vein,  Executive  Director 

Department  of  Telecommunications  &  Information  Services 
875  Stevenson  Street,  5^*"  Floor 
San  Francisco,  CA  94103 

Dear  Mr.  Vein: 

The  Controller's  Office,  City  Services  Auditor,  presents  its  audit  report  of  the  Department  of 
Telecommunications  and  Information  Services  (DTIS).  The  audit  objectives  were  to  assess 
the  quality  and  cost  of  DTIS'  services  to  determine  how  DTIS  can  improve  its  service  to 
clients  and  its  rates,  and  to  assess  the  quality  of  DTIS'  performance  measures  and  the 
accuracy  of  its  reported  results. 

We  concluded  that  DTIS  can  reduce  its  costs  and  improve  its  customer  service  by  simplifying 
the  rate  model  and  making  its  rates  more  equitable,  predictable,  and  transparent  for  its 
clients,  and  by  consolidating  overlapping  and  duplicate  functions  both  within  DTIS  and  with 
other  City  departments.  Implementing  generally  recognized  project  management  techniques 
would  help  DTIS  ensure  accountability  for  its  projects.  DTIS  can  also  improve  its  ability  to 
measure  its  successes,  make  informed  decisions,  and  identify  areas  needing  improvement 
by  developing  performance  measures  that  reflect  the  services  it  provides  and  by  establishing 
data  collection  and  reporting  mechanisms  to  accurately  reflect  its  performance. 

The  audit  includes  23  recommendations  for  DTIS  to  reduce  costs,  improve  the  quality  of 
services,  improve  the  accuracy  of  accounting  and  billing  methods,  eliminate  overlapping  or 
duplicate  services,  and  improve  overall  accountability  for  performance.  The  department's 
response  to  the  audit  is  attached  as  Appendix  B.  DTIS  has  agreed  with  most  of  the  audit 
recommendations  and  expressed  an  interest  in  making  changes  within  the  organization. 

We  appreciate  the  assistance  and  cooperation  that  DTIS  staff  and  staff  in  other  City 
departments  provided  to  us  during  the  audit. 
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City  and  County  of  San  Francisco 

Office  of  the  Controller  -  City  Services  Auditor 


The  Department  of  Telecommunications  and  Information  Services  Needs  to 
Simplify  Its  Rate  Model,  Improve  Customer  Service,  and  Revamp  Performance 
Measures 


Audit  No.  05053 
November  17,  2006 


Purpose  of  the  Audit 

We  assessed  the  quality  and  cost  of  services  provided  by  the  Department  of  Telecommunications  and 
information  Services'  (DTIS)  to  determine  if  it  can  improve  its  service  to  clients  and  its  rates.  We  also  reviewed 
DTIS'  performance  measures  to  determine  their  meaningfulness  and  the  accuracy  of  their  reported  results. 


Highlights 

DTIS'  rate  model  is  complex  and  costs  over  $1 .8  million  annually  to 
administer,  including  35,000  labor  hours  (17  PTEs)  and  the  nonlabor  costs  of 
maintaining  systems  to  support  the  budget  development,  rate  setting,  and 
client  billing  processes.  Despite  its  complexity  and  cost,  the  rate  model  does 
not  reflect  the  actual  cost  of  services  or  ensure  fair  and  accurate  billings  for 
the  $90  million  of  services  that  DTIS  provides  its  clients  annually. 

Clients  often  pay  different  rates  for  the  same  service  and  rates  differing  from 
those  published  in  its  Rate  Handbook.  For  example,  the  published  hourly 
rate  for  database  administration  services  was  $108.45  per  hour,  but  DTIS 
charged  10  of  the  15  departments  using  these  services  rates  ranging  from 
$112  to  $162.  The  discrepancy  between  the  published  rates  and  the  actual 
rates  paid  for  the  services  reviewed  totaled  over  $4  million,  with  some 
departments  being  undercharged  for  some  services  and  overcharged  for 
others.  As  a  result,  some  departments  subsidized  the  cost  of  service  to 
others  and  clients  could  not  control  or  predict  their  information  technology 
expenditures  by  changing  their  level  of  use. 

i  Clients  are  dissatisfied  with  the  cost  of  DTIS  services,  and  61  percent  (14  of 
1 23)  of  clients  responding  to  a  satisfaction  survey  would  not  continue  to  use 
I  DTIS'  services  if  given  autonomy  over  their  services.  Some  clients  have 
■  established  their  own  internal  IT  functions  or  purchased  IT  services  from 

other  sources,  thereby  increasing  costs  for  remaining  clients  and  causing  the 

City  to  pay  for  duplicate  services. 

DTIS  maintains  three  separate  help  desks,  at  a  cost  of  $1.2  million,  although 
previous  studies  of  DTIS  and  industry  standards  recommend  having  only  one 
help  desk  in  an  organization.  Consolidating  the  help  desks  would  achieve 
savings  and  improve  service  to  clients  by  eliminating  overlapping  duties  and 
responsibilities  and  allowing  DTIS  to  standardize  its  help  desk  policies  and 
procedures.  In  addition,  DTIS  provides  operator  services,  alarm  monitoring 
and  maintenance  functions,  and  website  services  that  are  also  provided  by 
other  city  departments  or  vendors. 

DTIS  reported  reliable  and  accurate  results  for  only  three  of  its  nineteen 
performance  measures,  and  only  ten  of  its  measures  were  meaningful  and 
useful.  Not  knowing  how  well  it  is  performing  in  key  areas  makes  it  difficult 
for  DTIS  to  measure  its  success  or  identify  areas  needing  improvement. 


Recommendations 

The  audit  report  includes  23 
recommendations  for  DTIS  to 
reduce  its  costs,  improve  its 
quality  of  service,  improve  the 
accuracy  of  its  accounting  and 
billing  methods,  eliminate 
overlapping  or  duplicate 
services,  and  improve  its 
overall  accountability  for 
performance.  Specifically, 
DTIS  should: 

•  Simplify  the  rate  model  so 
costs  are  more  easily 
identifiable  and  reflect  the 
actual  cost  of  service. 

•  Develop  a  customer  service 
evaluation  and  improvement 
plan  regarding  client 
satisfaction  with  its  services, 
costs,  and  billing  methods. 

•  Consolidate  the  help  desk, 
consolidate  operator  and 
monitoring  functions,  and 
eliminate  duplicate  website 
costs  to  save  around 

$1  million  annually  and 
improve  efficiency. 

•  Establish  performance 
measures  that  are 
meaningful  and  establish 
data  collection  and  reporting 
mechanisms  to  accurately 
reflect  performance. 


Copies  of  the  full  report  may  be  obtained  at: 
Controller's  Office  •  City  Hall,  Room  316  •  1  Dr.  Carlton  B.  Goodlett  Place  •  San  Francisco,  CA  94102  •  415.554.7500 
or  on  the  Internet  at  www.sfgov.org/controller 


INTRODUCTION 


BACKGROUND 

The  City  and  County  of  San  Francisco  (City)  has  made  several 
changes  in  the  past  decade  in  how  it  administers  information 
technology  (IT)  and  telecommunications.  In  1996,  the  City 
adopted  an  IT  Strategic  Plan,  developed  by  KPMG  Peat 
Marwick,  LLP,  which  became  the  foundation  for  the  City's 
current  IT  architecture.  The  plan  resulted  in  creation  of  the 
Department  of  Telecommunications  and  Information  Services 
(DTIS),  the  Committee  on  Information  Technology  (COIT),  and 
the  Strategic  Management  and  Planning  Group.  In  2005, 
DTIS,  the  Department  of  Public  Works,  and  Administrative 
Services  merged  to  create  the  General  Services  Agency.  At 
the  time  of  this  audit,  changes  in  DTIS'  leadership  and 
governance  structure  were  in  progress. 

DTIS'  Primary  Revenue  Source  Is  Client  Billings 

DTIS  is  an  internal  service  fund  department.  Internal  service 
funds  finance  and  account  for  operations  that  provide  goods  or 
services  to  other  governmental  units  on  a  cost-reimbursement 
basis.  The  goal  of  the  fund  is  to  measure  the  cost  of  providing 
goods  or  services  for  the  purpose  of  fully  recovering  that  cost 
through  fees  or  charges.  Governments  commonly  use  internal 
service  funds  for  a  wide  variety  of  activities,  including 
information  systems,  central  garages  and  motor  pools,  copying 
and  printing  services,  purchasing,  and  central  stores. 

DTIS'  fiscal  year  (FY)  2005-06  budget  included  88  percent  of 
its  revenues  from  departmental  work  orders,  which  are 
payments  from  clients  to  DTIS  for  services  rendered; 
10  percent  from  the  general  fund;  and  two  percent  from  cable 
franchise  fees.  DTIS  also  receives  "off  budget"  revenues 
during  the  year  for  additional  work  or  services.  DTIS  uses  a 
rate  model  that  it  developed  in  2000  to  calculate  the  rates  for 
billing  client  departments  and  recovering  its  costs.  DTIS  sets 
its  rates  during  the  annual  budget  process  and  publishes  them 
in  its  Rate  Handbook.  DTIS  distributed  its  FY  2005-06 
expenses  among  the  following  IT  and  telecommunication 
service  categories: 
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EXHIBIT  1 


Service  Categories  and  Expenses 


Total 

%  of 

OGrviCG  CSiGQOrV 

FY  2005-06 

Total 

Indir6ct  SBrvicss: 

Psss-throuQhs  snd  other  charges 

$17,756,924 

19.7 

Telephone  Billing 

15  058,784 

16.7 

Direct  Services: 

Applic3tions  3nd  d3t3b3se  3drninistr3tion  (DBA)  services 

18,209,689 

20.2 

Other  sp6ci3l  projects  3nd  citywide  initistives 

8,1 1 1 ,999 

9.0 

M3infr3me  services 

6,772,087 

7.5 

M3inten3nce  snd  support  for  public  safety  emergency  systems 

6,021 ,188 

6.7 

4,674,087 

5.2 

T^lecx>mrnunicm^^    consultin  for  clients 
e  ecommunica  ions  consu  ing  or  c  1  n 

3,517,733 

3.9 

Loc3l  Ares  Network  (LAN)  &  Wide  Area  Network  (WAN)  client  support 

3,392,054 

3.8 

1,784,832 

2.0 

WAN  mahten^nce 

1,661,127 

1.8 

SFGTV  (Government  Cable  Channels) 

867,444 

1.0 

697,170 

0.8 

Citywide  public  website  services 

554,960 

0.6 

Geographical  Information  System  (GIS) 

546,960 

0.6 

Network  planning,  design.  &  security 

427,737 

0.5 

Alarm  monitoring 

129,000 

0.1 

Citywide  technology  research  &  adoption 

99,867 

 oi 

Total 

$90,283,644 

100.0% 

Source  DTIS  revised  budget  as  of  3/27/06 

OBJECTIVES,  SCOPE,  AND  METHODOLOGY 

The  audit  objectives  were  to  assess  the  quality  and  cost  of 
DTIS'  services  to  determine  if  DTIS  can  improve  service  to 
clients  and  rates,  and  if  so,  how.  We  also  reviewed  the 
meaningfulness  of  the  department's  performance  measures 
and  the  reliability  of  their  reported  results. 

We  reviewed  three  service  categories: 

•  Applications  and  database  administration  (DBA) 

•  Mainframe 

•  Maintenance  and  support  for  public  safety  emergency 
systems  (public  safety  services) 

We  selected  these  services  because  they  range  across  a  wide 
spectrum  of  clients,  represent  54  percent  of  DTIS'  direct 


service  budget  and  34  percent  of  its  total  budget,  and  receive 
many  requests  for  service.  We  reviewed  DTIS'  rate  model 
(rate  setting  process  and  billing  methods)  because  of  its 
complexity  and  its  effect  on  service  quality  and  cost.  We  did 
not  review  all  citywide  information  technology  and 
telecommunication  services  or  the  security  of  the  City's 
information  and  telecommunications  systems.  Although  the 
focus  of  the  audit  was  not  to  determine  which  services  should 
be  centralized  or  decentralized,  the  audit  nevertheless  touches 
upon  this  subject,  particularly  where  necessary  to  address  how 
the  quality  and  cost  of  service  can  be  improved.  Although 
building  security  was  outside  the  scope  of  the  audit,  we 
addressed  this  issue  because  we  observed  that  one  of  DTIS' 
facilities  was  not  secured  and  was  susceptible  to  theft. 

The  audit  methodology  included: 

•  Examining  the  rate  setting  and  billing  process  by  reviewing 
three  years  of  billing  history 

•  Estimating  the  cost  of  administering  the  DTIS  rate  model 

•  Researching  IT  rate  models  and  other  similar  systems 

•  Assessing  DTIS'  performance  measures  to  gauge  their 
accuracy/reliability  and  usefulness/meaningfulness 

•  Conducting  customer  satisfaction  surveys 

•  Interviewing  DTIS  staff  and  management,  industry  experts, 
vendors,  and  knowledgeable  staff  in  other  City 
departments 

•  Observing  DTIS  operations  at  875  Stevenson,  901  Rankin, 
101 1  Turk,  and  the  Central  Radio  Station  (CRS)  at  Twin 
Peaks 

•  Benchmarking  best  practices  and  industry  standards  for 
mainframe,  applications  development,  and  database 
administration 

We  conducted  this  performance  audit  in  accordance  with 
Generally  Accepted  Government  Auditing  Standards.  We 
believe  that  we  adequately  planned  and  performed  the  audit  to 
achieve  the  audit  objectives  and  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives,  as  required  by  these  standards. 
The  standards  require  that  the  audit  organization  be  free  both 
in  fact  and  in  appearance  from  organizational  impairments  to 
independence.  COIT  was  established  as  part  of  the 
Controller's  Office  and  had  governance  responsibility  for  some 
of  the  City's  IT  decisions.  Because  we  did  not  review  COIT,  its 
functions,  or  its  relationship  with  DTIS  and  the  audit  was 
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limited  to  DTIS,  our  independence  was  not  impaired. 
However,  we  are  disclosing  this  relationship  to  comply  with  the 
requirement  to  disclose  situations  that  some  may  perceive  as 
a  possible  impairment. 


Staff:    Pablo  Federico,  Audit  Manager 
Alexandra  Bidot,  Audit  Staff 
Randle  McClure,  Audit  Staff 


CHAPTER  1  -  COST  AND  SERVICE  QUALITY 


DTIS'  rate  model  is  complex  and  costs  over  $1 .8  million 
annually  to  administer,  including  the  cost  of  billing  departments 
for  services  provided.  Despite  its  complexity  and  cost,  the  rate 
model  does  not  reflect  the  actual  cost  of  services,  ensure  fair 
and  accurate  billings  for  the  $90  million  of  services  that  DTIS 
provides  its  clients  annually,  or  allow  clients  to  control  their  IT 
spending  by  changing  their  use  of  DTIS'  services. 

As  a  result,  clients  are  dissatisfied  and  61  percent  (14  of  23)  of 
clients  responding  to  a  customer  satisfaction  survey  would  not 
continue  to  use  DTIS'  services  if  given  autonomy  over  their  IT 
services.  Some  clients  have  established  their  own  internal  IT 
functions  or  purchased  IT  services  from  vendors  -  in  some 
cases  while  continuing  to  pay  DTIS  for  the  same  service.  This 
has  resulted  in  inefficient  use  of  DTIS  and  City  resources, 
duplicate  costs,  and  increased  costs  for  IT  services. 

Finding  1.1.  Complex  Rate  IVIodel  Costs  Over 
$1.8  Million  but  Does  Not  Reflect  Actual  Costs 

The  rate  model's  $1.8  million  administrative  cost  includes 
35,500  labor  hours  (17  PTEs)  and  the  nonlabor  costs  of 
maintaining  systems  to  support  the  budget  development,  rate 
setting,  and  client  billing  processes.  DTIS'  clients  absorb  this 
cost  through  the  rates.  This  level  of  effort  and  cost  are  not 
justified  because: 

•  Clients  often  do  not  pay  published  rates;  billings  do  not 
reflect  actual  units  of  service  provided. 

•  Clients  often  cannot  control  costs  by  changing  their  use. 

•  Rates  do  not  reflect  actual  cost  of  service,  which  is  the 
basic  purpose  of  an  internal  service  fund.  Instead,  the  rate 
model  functions  as  a  tool  to  ensure  that  DTIS  recovers  its 
budgeted  expenditures  rather  than  to  accurately  reflect  the 
fixed  and  variable  costs  associated  with  each  specific 
service. 

•  Clients  are  dissatisfied  with  cost  and  the  rate  model. 

Clients  Often  Do  Not  Pay  Published  Rates 

The  rates  that  clients  actually  pay  often  differ  from  the 
published  rates.  As  a  result,  clients  pay  different  rates  for  the 
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same  service  and  subsidize  the  cost  of  service  to  others,  and 
DTIS  over-  or  underrecovers  its  costs  of  services.  Clients  do 
not  see  that  they  are  not  paying  the  published  rate  because 
their  billings  always  reflect  the  correct  rate;  DTIS  adjusts  the 
number  of  service  units  provided. 

For  some  services,  DTIS  calculates  the  billings  by  multiplying 
the  current  rate  by  the  units  of  service  provided  in  an  earlier 
fiscal  year,  usually  one  or  t\NO  years  earlier.  However,  DTIS 
did  not  charge  the  correct  rate  for  some  clients  according  to 
the  calculation  methods  they  developed.  The  following 
examples  of  DTIS"  charges  to  clients  for  printing  costs  in 
FY  2003-04  and  FY  2004-05  demonstrate  some  of  the 
discrepancies  between  the  published  prices  and  the  actual 
prices  paid: 


Printing  Charges 


Adult  Probation 

Controller's  Office 

Trial  Courts 

FY  2003-04 

FY  2004-05 

FY  2003-04 

FY  2004-05 

FY  2003-04 

FY  2004-05 

'Lines  of  Print 

28.336 

19,210 

2,769,754 

1,700,073 

231,302 

243,172 

Published  Rate 

X  $0,235 

X  $0.25 

X  $0,235 

X  $0.25 

X  $0,235 

x$0.25 

Expected  Bill 

$6.659 

$4,803 

$650.892 

$425018 

$54.356 

$60.793 

Actual  Bill 

$41,852 

$44,164 

$468,732 

$305,979 

$54,356 

$30,397 

Over/(under)-charge 

$35,193 

$39,361 

($182,160) 

($119,039) 

Accurate  bill 

($30,397) 

Actual  Rate  Paid 

$1.47 

$2.30 

$0.18 

$0.17 

$0,235 

$0,125 

Adult  Probation  was 
overcharged  in  both  years. 
Although  the  lines  of  print 
decreased  in  FY  2002-03, 
the  corresponding  billing  for 
FY  2004-05  increased.  The 
slight  increase  in  the  rate  in 
FY  2004-05  was  insufficient 
to  account  for  the  billing 
increase,  as  shown  by  the 
significantly  higher  rate  that 
Adult  Probation  actually 
paid. 

The  Controller's  Office  was 
undercharged  in  both  years. 
The  billing  decreased  as 
expected  in  FY  2004-05 
based  on  the  fewer  lines  of 
print  in  FY  2002-03. 
However,  the  rate  the 
Controller's  Office  actually 
paid  shows  that  the  billing 
decreased  more  than  it 
should  have  since  the 
published  rate  increased. 

The  Trial  Courts  were 
accurately  billed  in 
FY  2003-04,  but  were 
undercharged  in 
FY  2004-05.  Although  the 
printed  lines  increased  in 
FY  2002-03,  the  billing 
decreased  in  FY  2004-05, 
as  did  the  actual  rate  paid. 

•  Charges  are  based  on  a  rate  per  100  lines  of  print  (i  e.,  quantities  are  expressed  per  100  lines  of  print).  Bills  are 
based  on  the  number  of  lines  printed  two  years  prior  (e  g  ,  the  number  of  lines  printed  in  FY  2001-02  are  used  to 
calculate  the  bills  for  FY  2003-04) 


Source   DTIS  billing  system  charges  and  mainframe  records  for  lines  of  print 

DTIS  made  similar  adjustments  for  other  services  that  we 
reviewed.  For  example,  the  published  rate  for  DBA  work  in 
FY  2004-05  was  $1 08.45  per  hour,  but  DTIS  charged  1 0  of  the 
15  departments  using  these  services  different  hourly  rates 
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ranging  from  $112  to  $162.  The  discrepancy  between  the 
published  rates  and  the  actual  rates  paid  for  the  three  services 
reviewed  (mainframe,  applications  development  and  DBA,  and 
public  safety  services)  totaled  over  $4  million,  with  some 
departments  being  undercharged  for  some  services  and 
overcharged  for  others. 

These  examples  demonstrate  that  the  cost  to  develop  and 
administer  the  rate  model  is  not  justified  because  DTIS  adjusts 
client  billings  as  needed  to  recover  its  costs  without  regard  to 
the  effect  it  has  on  the  actual  rates  that  clients  pay. 

Reasons  why  DTIS  makes  these  adjustments  vary: 

•  Changes  to  DTIS'  and  departments'  budgets  sometimes 
occur  after  the  budgets  are  balanced  and  submitted, 
resulting  in  revenues  and  expenditures  being  out  of 
balance.  The  final  rates  are  based  on  the  budget 
submitted  to  the  Board  of  Supervisors  for  review,  rather 
than  on  the  final  adopted  budget. 

•  Departments  sometimes  do  not  have  enough  money  to 
pay  the  actual  cost  of  services  provided  during  the  year  or 
underestimate  their  budget  for  services.  DTIS  makes 
internal  adjustments  to  recover  costs,  including  shifting  the 
costs  to  other  product  lines  or  to  overhead. 

•  DTIS  bills  project  planning  costs  to  overhead  when  funding 
for  the  work  is  not  yet  received.  DTIS  redistributes  these 
costs  through  the  rate  model  making  them  difficult  to  track. 

•  DTIS  redistributes  some  of  its  costs  between  services. 
Division  managers  are  allowed  to  trade  costs  between 
services  or  projects  as  they  deem  appropriate,  creating  a 
lack  of  transparency  in  billing  and  accounting. 

Clients  Often  Cannot  Control  Costs  by  Changing  Their 
Use 

Clients  cannot  control  or  predict  their  IT  expenditures  by 
changing  their  level  of  use  because  of  DTIS'  adjustments  to 
their  billings.  As  shown  in  the  examples,  a  client  may  reduce 
its  level  of  use  from  one  year  to  the  next,  but  its  cost  can 
actually  increase.  The  adjustments  make  it  difficult  for  clients 
to  confirm  the  accuracy  or  appropriateness  of  DTIS'  charges 
and  to  assess  whether  DTIS'  rates  are  competitive  or 
reasonable  compared  to  other  service  providers. 
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Clients  also  have  little  to  no  control  over  their  charges  for 
fixed-cost  services  such  as  enterprise-wide  services  and 
infrastructure  work, '  DTIS  allocates  these  costs  among  its 
clients,  but  the  costs  are  not  directly  attributable  to  usage  or 
benefit.  For  example,  infrastructure  work  usually  benefits 
multiple  departments  and  must  be  done  whether  it  is 
separately  budgeted  or  not.  DTIS  allocates  infrastructure 
costs  across  its  product  lines  although  the  costs  are  not 
affected  by  client  consumption. 

Industry  literature  recommends  charging  clients  only  for 
unshared  IT  expenses  that  clients  can  control  through  their 
rate  of  use.  Also,  a  report  of  DTIS'  compliance  with  ITIL^ 
standards  recommended  that  DTIS  use  the  rate  model  only  for 
work  that  clients  can  influence  through  their  consumption. 
This  practice  assumes  that  users  have  control  over  their  own 
use  and  will  therefore  make  better  decisions  when  faced  with 
the  costs  of  the  actual  level  of  IT  services  consumed.  Despite 
the  standard,  DTIS  includes  these  costs  in  the  rates  because 
of  reduced  general  fund  support  for  IT  infrastructure  over  the 
last  few  years  and  the  expectation  for  DTIS  to  fully  recover  its 
costs  of  operations. 

Rate  Model  Does  Not  Reflect  Actual  Costs 

DTIS  cannot  ensure  that  it  fully  recovers  its  cost  of  services 
through  the  rates  because  the  rate  model  is  based  on  DTIS' 
budgeted  expenditures  rather  than  the  actual  cost  of  service. 
Although  DTIS  has  historically  recovered  its  total  costs 
because  of  the  adjustments  it  makes  to  client  billings,  it  has 
over-  and  underrecovered  its  budgeted  expenditures  for 
specific  services.  For  example,  by  adjusting  client  billings  in 
FY  2004-05,  DTIS  recovered  $1.8  million  less  for  mainframe 
services  but  recovered  $82,000  more  for  DBA  services  than  it 
should  have  recovered  for  these  services.  As  a  result,  the  rate 
model  does  not  meet  the  requirements  for  internal  service 
funds  to  improve  financial  management  and  fully  recover  the 
cost  of  services. 


'  Enterprise  services  are  products  or  services,  such  as  e-mail,  that  are  provided  or  administered  centrally  for 
multiple  users  Infrastructure  work  includes  the  labor,  hardware,  and  software  necessary  to  support  the  City's  IT 
and  telecommunications  services 

'  ITIL  (the  Information  Technology  Infrastructure  Library)  is  a  framework  of  best  practices,  drawn  from  the  public 
and  private  sectors,  that  facilitate  the  delivery  of  high  quality  IT  services  ITIL  outlines  an  extensive  set  of 
management  procedures  to  achieve  quality  and  value  for  money  in  IT  operations. 
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DTIS  bills  its  clients  for  the  number  of  units  of  sale  (e.g.,  per 
hour,  per  number  of  devices  maintained)  provided  for  a  service 
or  through  a  monthly  fee,  but  tracks  its  costs  by  division.  The 
units  of  sale  do  not  always  reflect  how  DTIS  incurs  costs  for  a 
service  because  some  divisions  provide  multiple  types  of 
services  and  some  services  overlap  divisions.  The  mix  of 
fixed  and  variable  costs  differ  for  a  division  and  a  service  and 
can  be  affected  by  the  number  of  clients  that  the  costs  can  be 
distributed  among  and  the  allocation  of  indirect  costs.  For 
example,  DTIS  charges  public  safety  clients  for  wiring  work 
based  on  the  number  of  fire  alarm  and  police  call  boxes,  but 
the  cost  of  service  is  based  on  the  labor  and  materials  to 
install  or  relocate  these  devices.  Since  DTIS  does  not  track  its 
costs  for  installing  or  relocating  each  device,  it  cannot 
reconcile  division  costs  with  the  actual  costs  incurred  or 
assess  the  reasonableness  of  the  rate  for  the  service. 

Clients  Dissatisfied  With  Cost  and  Rate  Model 

Clients  are  not  satisfied  with  the  cost  of  DTIS  services  and 
with  its  rate  model  and  61  percent  (14  of  23)  of  clients 
responding  to  our  customer  satisfaction  survey  would  not 
continue  to  use  DTIS'  services  if  given  autonomy  over  IT 
services.  In  addition, 

•  58  percent  (14  of  24)  of  the  respondents  are  not  satisfied 
with  the  billing  detail  that  DTIS  provides 

•  67  percent  (1 6  of  24)  of  the  respondents  do  not 
understand  the  rate  setting  and  billing  methods 

•  Clients  perceive  that  DTIS'  rates  are  neither  competitive 
nor  cost-effective,  and  92  percent  (22  of  24)  of  the 
respondents  believe  that  the  same  services  are  available 
elsewhere  for  a  lower  cost 

•  Over  80  percent  (1 9  of  23)  of  the  respondents  believe  that 
DTIS  can  improve  its  cost  of  service  by  increasing  its 
accountability  to  clients,  lowering  overhead  costs, 
changing  the  funding  structure,  and  reducing  its  scope  of 
operations 

Clients  have  sought  services  from  other  sources  in  part 
because  they  are  dissatisfied  with  the  rate  model  and  DTIS' 
costs,  accounting,  and  billing  methods.  Some  departments 
have  purchased  IT  services  such  as  mainframe  and  website 
services  from  vendors  instead  of  from  DTIS,  while  other 
departments  have  increased  their  internal  IT  capacities.  This 
has  increased  the  costs  for  remaining  clients  and  has 
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contributed  to  duplicate  costs  for  IT  and  telecommunication 
services  across  the  City. 

The  City  has  an  interest  in  maintaining  certain  services 
internally  because  it  can  achieve  cost  effectiveness,  increase 
system  security,  and  maintain  IT  and  telecommunication 
standards  for  City  systems.  In  some  cases,  departments' 
decisions  to  use  vendors  increased  costs  for  remaining  clients 
because  the  City  invested  in  citywide  solutions  which  have 
fixed  costs  that  are  allocated  among  the  clients.  Security 
issues  may  also  arise  as  departments  rely  on  a  decentralized 
matrix  of  vendors  for  their  IT  and  telecommunications  services 
that  may  not  use  the  same  degree  of  redundancy  and  security 
as  City  systems.  Additionally,  establishing  and  maintaining 
standards  for  IT  and  telecommunications  is  difficult  when 
departments  use  a  variety  of  vendors  and  technologies  that 
may  lead  to  system  incompatibilities  and  inhibit  the  City's 
business  processes. 

Recommendations: 

1 .  DTIS  should  simplify  the  rate  model  and  restructure  it  to 
reflect  the  actual  cost  of  services: 

•  Reduce  the  number  of  product  lines 

•  Structure  the  units  of  service  to  reflect  how  costs  are 
incurred  (e.g.,  change  the  unit  of  service  for  public 
safety  wiring  work  from  number  of  devices  to  an  hourly 
rate)  and  to  allow  clients  to  control  their  costs  based  on 
their  use  of  services 

•  Identify  and  explain  the  variable  and  fixed  costs  for 
each  service  in  the  Rate  Handbook 

2.  DTIS  should  remove  from  the  rate  model  the  costs  of 
infrastructure  work  and  enterprise  services  that  are  not 
affected  by  clients'  level  of  use.  DTIS  should  request  a 
separate  budget  allocation  for  these  services  and  revise  its 
rate  model  accordingly. 

3.  DTIS  should  track  the  costs  to  provide  each  service  and 
update  the  rate  model  annually  to  reflect  actual  costs.  This 
should  include  developing  policies  and  procedures  for  time 
entry  and  cost  allocation  and  should  allow  clients  to 
understand  the  relationship  among  budgeted  work, 
percentage  of  work  complete,  and  billing  for  each  project. 
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4.  DTIS'  billings  should  reflect  the  actual  number  of  units  of 
service  provided  to  ensure  clients  receive  fair  and  accurate 
bills. 

5.  Before  accepting  an  off-budget  project,  DTIS  should 
identify  its  funding  source,  how  its  cost  will  impact  the 
allocation  of  costs  among  city  departments,  and  whether 
resources  will  have  to  be  reallocated  from  existing  projects. 

Finding  1.2.  Customer  Satisfaction  Has  Declined; 
DTIS  Needs  to  Measure  Satisfaction 

DTIS  has  not  addressed  dissatisfaction  issues  identified 
through  client  surveys  performed  in  2001  and  2002,  and 
responses  to  a  survey  we  conducted  showed  that  customer 
satisfaction  has  declined: 

•  61  percent  (14  of  23)  of  respondents  would  not  continue  to 
use  DTIS  if  given  autonomy  over  IT  services 

•  Over  two-thirds  (17  of  24)  believe  that  IT  services  provided 
elsewhere  would  be  of  better  quality  than  what  DTIS 
provides 

A  consultant  survey  of  DTIS'  clients  in  2001  stated  that  DTIS  is 
a  technology  laggard  and  poor  service  provider  and  that  these 
conditions  had  become  a  major  barrier  to  its  success.  The 
report  stated  that  it  was  critical  for  DTIS  to  address  these 
issues  "broadly  and  forcefully."  A  year  later,  COIT  conducted 
a  client  survey  and  graded  DTIS'  overall  service  delivery  as 
acceptable  and  meeting  expectations. 

Clients  responding  to  our  survey  were  slightly  less  satisfied 
with  DTIS'  quality  of  service  than  they  were  at  the  time  of  the 
2002  COIT  survey.  All  but  one  of  the  respondents  stated  that 
DTIS  could  improve  its  quality  of  service.  Clients  responded 
that  holding  DTIS  more  accountable  to  its  clients  would  be  the 
most  effective  solution  for  improving  DTIS'  quality  of  service. 

Clients  were  more  satisfied  with  the  quality  of  three  specific 
services  that  we  surveyed  -  public  safety,  mainframe,  and 
applications  development  and  DBA -than  with  DTIS'  services 
overall.  They  provided  the  following  positive  responses: 

•  Public  safety  service  clients  were  satisfied  with  the  work 
that  DTIS  staff  perform  and  their  responsiveness 
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•  Mainframe  clients  were  satisfied  with  the  consistency  and 

reliability  of  the  mainframe  information 

•  Applications  development  and  DBA  clients  were  satisfied 
with  the  quality  of  service,  responsiveness  of  DTIS  staff, 
and  DTIS'  ability  to  complete  their  specific  projects  on  time 
and  within  budget 

Despite  these  positive  comments,  clients  offered  specific 
suggestions  for  improvements  within  these  services  such  as: 

•  Cross-train  public  safety  services  employees  within  the 
various  functional  units  to  achieve  efficiency  and  lower 
costs 

•  More  frequently  update  the  records  of  whom  to  contact 
when  a  building  alarm  goes  off 

•  Adhere  to  the  E91 1  service  desk's  escalation  procedures 
(i.e.,  measures  for  handling  higher  priority  requests) 

Clients  expressed  dissatisfaction  with  the  following: 

•  Billing  and  costs  (discussed  in  Finding  1.1) 

•  Lengthy  delays  with  support  and  projects  (other  than  those 
for  applications  development  and  DBA  sen/ices) 

•  Mainframe  help  desk  operators  lack  of  knowledge  of  where 
to  send  clients  for  assistance 

Despite  multiple  surveys  showing  a  need  for  improvement, 
DTIS  has  not  established  a  mechanism  to  measure  overall 
customer  satisfaction  or  satisfaction  with  specific  services.  It 
does  not  conduct  its  own  surveys,  conduct  project-specific 
evaluations,  or  have  a  system  to  track  customer  complaints  by 
type.  DTIS'  2005  Efficiency  Plan  set  a  goal  to  conduct  an 
annual  benchmarking  survey  on  customer  satisfaction,  but 
DTIS  has  not  yet  done  so.  Without  measuring  customer 
satisfaction,  DTIS  will  not  be  able  to  focus  on  its  weaknesses 
and  take  corrective  action  in  the  areas  that  warrant  attention. 
Without  corrective  action,  clients  are  likely  to  seek  IT  services 
elsewhere. 

Recommendations: 

6    DTIS  should  develop  a  customer  service  evaluation  and 
improvement  plan  with  actions  for  improving  satisfaction 
with  services,  costs,  and  accounting.  The  plan  should 
include  measurable  objectives  and  actions  to  improve 


client  satisfaction  and  ways  that  DTIS  will  advertise  its 
successes  to  clients. 

7.  DTIS  should  survey  customer  satisfaction  with  services, 
costs,  and  accounting  at  least  annually  in  order  to  gauge 
success  or  areas  needing  improvement.  The  survey 
results  should  be  incorporated  into  the  customer  service 
improvement  plan  and  reported  annually  through  the 
budget  process. 

8.  DTIS  should  use  the  results  of  its  customer  satisfaction 
surveys  to  justify  additional  enterprise  ser\/ices  that  it 
proposes  to  take  on.  Additional  responsibilities  for  DTIS  in 
citywide  IT  and  telecommunications  services  should  be 
dependent  on  its  capacity  and  historical  performance  but 
should  also  be  based  on  client's  satisfaction  and  trust  in 
DTIS'  capacities. 

Finding  1.3.  Consolidate  Functions  to  Save 

$1  Million  Annually  and  Eliminate  Duplicate  Costs 

DTIS  can  save  approximately  $729,000  annually  and  improve 
its  efficiency  and  customer  service  by  consolidating  its  three 
help  desk  functions  and  by  consolidating  or  eliminating 
operator  and  monitoring  functions  that  overlap  or  duplicate 
those  provided  by  other  departments.  The  City  can  save  over 
$285,000  by  eliminating  duplicate  costs  for  website  services 
that  departments  purchase  from  vendors  or  provide  with  their 
own  staff. 

Consolidate  Three  Help  Desks  to  Save  $729,000  Annually 
and  Improve  Customer  Service 

Consolidating  its  help  desks  would  improve  customer  service 
and  eliminate  the  need  for  ten  positions,  which  would  save 
DTIS  approximately  $729,000  annually.  DTIS  has  three 
separate  help  desk  functions  that  cost  $1 .2  million  annually  for 
labor,  including  thirteen  1022  IS  Administrator  II  positions  and 
two  1024  IS  Administrator-Supervisor  positions  distributed  as 
follows: 

•    The  general  help  desk  tracks  incoming  sen/ice  requests 
and  responds  to  general  service  questions;  it  is  staffed  by 
two  employees  and  a  manager 
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•  The  telecom  help  desk  responds  to  issues  related  to 

equipment  and  service  moves,  adds,  and  changes;  it  is 
staffed  by  three  employees 

•  The  E91 1  service  desk  supports  the  public  safety 
emergency  systems  and  other  public  safety  requests  and 
operates  24-hours  a  day;  it  is  staffed  by  six  employees, 
one  manager,  and  two  part-time  employees 

We  estimate  that  DTIS  could  reasonably  operate  a 
consolidated  help  desk  function  with  four  1022s  and  one  1024 
manager.  This  estimate  is  based  on  a  formula  developed  by 
Microsoft  specifically  for  estimating  staffing  needs  of  a  help 
desk.  We  reviewed  the  calls  and  incidents  of  the  general  help 
desk,  telecom  help  desk,  and  the  E91 1  service  desk  and 
estimated  that  together,  the  help  desks  manage  about  36 
incidents  per  day.  We  assumed  that  the  average  time  a  help 
desk  employee  spends  per  incident  is  30  minutes.  This  is  a 
high  estimate  considering  that  other  jurisdictions  reported 
spending  12  and  15  minutes  per  incident.  However,  in  order 
to  provide  a  reasonable  estimate  of  staffing  needs,  we  used  30 
minutes  for  our  calculation.  We  assumed  a  utilization  rate 
(percentage  of  time  resolving  incidents)  of  75  percent.  Finally, 
we  estimated  78  percent  availability  based  on  actual 
department  leave  time  during  the  first  half  of  FY  2005-06.  Our 
calculation  is  shown  below: 

Incidents  (36)  *  Average  Time  Spent  Resolving  an  Incident  (0.5  hours)  18  hours 

    =   =  3.85  persons 

Available  Time  (8  hours)  *  Utilization  Rate  (75%)  *  Availability  (78%)         4.68  available  hours 

Our  estimate  does  not  include  24-hour  staffing  for  the  E91 1 
service  desk  because  the  after-hours  service  desk  workload 
indicates  that  this  function  could  be  consolidated  with  its  other 
24-hour  monitoring  and  dispatching  services: 

•  73  percent  (1 0,01 0  of  1 3,691 )  of  all  incidents  came 
between  the  hours  of  7:00  a.m.  and  6:00  p.m.  from 
October  2003  through  April  2006 

•  Only  164  (0.01  percent)  of  all  incidents  were  priority  1 , 
which  are  cntical  incidents  that  must  be  addressed 
immediately 

•  Only  27  of  the  priority  1  tickets  were  opened  between  6:00 
p.m.  and  7:00  a.m.,  which  equates  to  one  after-hours 
priority  1  incident  every  34  days 

Consolidation  would  improve  customer  service  and  DTIS' 
ability  to  achieve  the  help  desk  industry  standard  of  "first  call 
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resolution"  by  eliminating  some  of  the  need  to  transfer  calls  for 
resolution.  Currently,  clients  contact  the  help  desk  expecting 
resolution,  but  the  issues  are  frequently  addressed  by  other 
non-help  desk  DTIS  staff.  For  example,  from  January  2005 
through  March  2006: 

•  Only  39  percent  (2,879  of  7,360)  of  the  calls  to  the  general 
and  telecom  help  desk  were  assigned  to  the  help  desk 
staff.  The  other  61  percent  were  assigned  to  other  DTIS 
staff  for  resolution. 

•  Only  41  percent  (3,648  of  8,823)  of  the  incidents  opened 
by  the  E91 1  service  desk  were  assigned  to  the  service 
desk  staff.  The  other  59  percent  were  assigned  to  other 
staff  for  resolution. 

•  Calls  are  also  forwarded  from  one  help  desk  to  another. 
For  example,  the  E91 1  service  desk  received  352  calls  that 
it  forwarded  to  the  general  help  desk. 

Redirecting  calls  from  one  help  desk  to  another  or  assigning 
them  elsewhere  contributed  to  DTIS  reporting  that  it  resolved 
only  33  percent  of  its  general  help  desk  calls  within  the  first 
call  resolution  standard  during  FY  2005-06.  DTIS  does  not 
track  the  percentage  of  calls  answered  within  the  standard  for 
the  E91 1  service  desk. 

Consolidating  the  help  desks  would  also  achieve  internal 
efficiency  and  improve  service  to  clients  by  eliminating 
overlapping  duties  and  responsibilities  and  allowing  DTIS  to 
standardize  its  help  desk  policies  and  procedures.  Currently, 
DTIS  does  not  track  the  aggregate  efforts  of  all  help  desks  and 
does  not  produce  reports  regarding  the  types  of  calls  and 
problems  that  would  allow  it  to  achieve  efficiencies.  Service 
desk  consolidation  would  allow  DTIS  to  standardize  its 
tracking  tools  and  reports  so  it  can  evaluate  the  number  and 
types  of  calls  received  and  calculate  the  true  cost  of  providing 
help  desk  support  to  clients. 

Consolidating  the  help  desks  would  achieve  compliance  with 
industry  standards  which  recommend  help  desk  consolidation 
to  achieve  capital  cost  savings  and  IT  labor  efficiencies,  and  to 
increase  service  levels  to  users.  An  ITIL  compliance  report 
issued  in  2004  recommended  that  DTIS  integrate  its  help 
desks  to  create  a  single  point  of  contact  for  all  clients  and  to 
bring  staff  to  a  skilled  level  within  six  months.  Peregrine, 
creator  of  the  help  desk  software  that  DTIS  uses,  issued  a 
white  paper  in  October  2005,  which  concluded  that  service 
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desk  consolidation  is  not  just  a  good  idea,  but  a  necessity  and 
economic  imperative  that  will  grow  more  critical  over  time. 
DTIS'  three  help  desks  and  the  multiple  points  of  contact  run 
contrary  to  industry  standards,  which  recommend  a  central 
point  of  contact  for  all  questions  or  service  requests  and  to 
track,  route,  and  respond  in  a  prioritized  and  systematic  way. 

The  actual  savings  will  depend  on  DTIS'  final  assessment  of 
how  to  consolidate  and  staff  the  help  desk  function.  However, 
our  estimate  shows  that  DTIS  can  achieve  savings  of 
approximately  $729,000  and  maintain  appropriate  staffing 
levels  for  a  consolidated  help  desk  function. 

Consolidate  Overlapping  Operator  and  Monitoring 
Functions 

The  City  could  achieve  efficiencies  and  reduce  its  costs  by 
consolidating  some  of  DTIS'  operator  and  monitoring 
functions.  The  amount  of  savings  will  be  determined  by  how 
DTIS  restructures  its  operator  and  monitoring  functions  and 
the  number  of  functions  it  can  move  to  other  departments. 
Additionally,  the  City  is  embarking  on  a  comprehensive  3-1-1 
system  which  presents  an  opportunity  for  DTIS  and  City 
departments  to  consolidate  their  operator  and  monitoring 
functions  to  maximize  efficiency  and  reduce  costs. 

Multiple  DTIS  employees  perform  operator  functions  that  are 
similar  to  functions  performed  by  other  departments.  DTIS 
has  two  operators  who  answer  the  City's  main  telephone  line 
and  DTIS'  main  line  during  business  hours.  DTIS  also  has 
one  person  at  the  Central  Fire  Alarm  Station  (CFAS)  at 
101 1  Turk  who  answers  the  City's  main  number  and  receives 
other  types  of  calls  after  hours,  including  calls  for: 

•  PBX^-related  issues 

•  Assistance  with  elevators  that  are  out  of  service 

•  Construction  emergencies,  construction  noise  complaints, 
and  other  related  issues  after  3:00  p.m.  for  the  Public 
Utilities  Commission  and  the  Department  of  Public  Works 
(DPW) 

•  Traffic  signal  complaints  after  3:00  p.m.  and  on  weekends 

and  holidays 


Short  for  pnvate  branch  exchange.  PBXs  are  private  telephone  networks  used  within  an  enterprise. 
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other  City  departments  provide  similar  operator  functions.  For 
example,  DPW's  Bureau  of  Street  Environmental  Services 
(BSES)  has  a  24-hour  service  line  that  the  Public  Utilities 
Commission  Wastewater  Enterprise  now  also  uses  for  after- 
hours  and  holiday  calls  dealing  with  sewer  problems.  These 
sewer-related  problems  were  previously  handled  by  the  DTIS 
CPAS  group. 

Overlap  also  occurs  in  the  systems  monitoring  roles  and 
responsibilities  of  the  DTIS  Public  Safety  Services  Division 
because  two  groups,  CPAS  and  System  Watch,  monitor  some 
of  the  same  systems.  Areas  of  overlap  include  testing 
community  security  cameras  and  monitoring  the  City's 
Auxiliary  Water  Supply  System  and  the  Outdoor  Public 
Warning  System  (formerly  known  as  the  SIRENS  system). 

DTIS  has  added  system  monitoring  roles  and  responsibilities 
without  determining  whether  these  services  are  provided  in  the 
most  efficient  and  economical  manner.  While  overlapping 
functions  might  create  redundancies  that  help  prevent 
incidents  being  missed,  the  overlap  should  be  planned, 
documented,  and  evaluated  frequently  rather  than  occurring 
haphazardly  as  technology,  systems,  and  needs  change. 

Clients  suggested  that  DTIS  improve  its  service  and  lower  its 
costs  by  consolidating  its  operator  and  monitoring  functions 
and  by  cross  training  employees  within  the  various  Public 
Safety  Services  groups  (CPAS,  System  Watch,  and  E91 1 
service  desk).  DTIS  has  recognized  the  need  for  integration 
and  cross  training  and  has  made  some  improvements, 
including  job  rotation  and  alternate  work  schedules  among  the 
systems  watch  staff.  However,  more  consolidation  and 
integration  can  occur  in  duties  and  responsibilities,  as  well  as 
in  systems  monitoring  and  tracking. 

DTIS  Can  Lower  City  Costs  by  Monitoring  and  Maintaining 
All  Alarms  for  City  Buildings 

DTIS  monitors  and  maintains  fire  and  burglar  alarms  for 
various  City  buildings,  but  some  departments  pay  vendors  for 
the  same  type  of  work.  We  could  not  determine  the  extent  to 
which  departments  pay  vendors  for  alarm  monitoring  and 
maintenance  because  the  City  does  not  maintain  a  central 
listing  of  amounts  paid  for  these  specific  sen/ices.  Some  of 
the  work  might  be  included  in  other  security-related  contracts 
such  as  security  guard  services.  Additionally,  the  amounts 
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paid  for  monitoring  versus  maintenance  work  are  not  easily 
distinguishable  and  vary  by  contract. 

According  to  DTIS,  vendors  that  provide  alarm  monitoring  for 
City  departments  contact  the  Police  or  Fire  Department 
through  the  City's  dispatchers.  Both  DTIS  and  other 
department  staff  agreed  that  it  makes  sense  for  DTIS  to 
monitor  all  City  alarms  to  achieve  operational  efficiencies. 

Despite  potential  savings  for  the  City  of  having  DTIS  monitor 
and  maintain  all  alarms,  there  are  certain  limitations.  In  some 
cases,  switching  the  alarms  so  that  they  are  monitored  by 
DTIS  can  be  done  without  much  work.  However,  in  other 
cases,  an  investment  needs  to  be  made  for  the  transition  from 
vendor  to  DTIS  for  monitoring  service.  The  cost  of  the 
transition  would  depend  on  the  internal  wihng  of  the  buildings 
and  the  condition  that  the  vendors  leave  them  in  after  pulling 
out  the  existing  alarms. 

Even  if  DTIS  is  able  to  take  on  more  alarm  monitoring  duties,  it 
is  limited  in  its  capacity  to  take  on  more  alarm  maintenance 
responsibilities.  In  a  survey  of  DTIS  alarm  clients,  most  were 
satisfied  with  the  monitoring  and  maintenance  work,  but  some 
indicated  that  the  DTIS  group  that  provides  the  maintenance 
work  was  already  stretched  in  its  responsibilities. 

Departments  Pay  Both  DTIS  and  Others  for  Website 
Services;  City  Can  Save  Over  $285,000 

The  City  could  save  over  $285,000  of  its  annual  contract  and 
labor  costs  if  all  departments  hosted  and  maintained  their 
websites  through  DTIS.  DTIS  maintains  a  city-affiliated 
website  on  www. sf gov. org  for  each  department/agency,  but  24 
departments/agencies  have  only  a  home  page  which  directs 
visitors  to  a  website  outside  www. sf gov. org.  Of  these  24 
departments/agencies,  12  can  be  considered  sufficiently 
distinct  to  justify  having  an  external  website  (e.g.,  SFO  Airport, 
various  museums)." 

The  remaining  12  departments  pay  DTIS  for  presence  on 
www.sfgov.org  and  also  pay  other  parties  (vendors  or  the 
City's  Reproduction  Services  within  the  General  Services 
Agency)  or  their  internal  staff  to  maintain  and  support  their 

*  DTIS  stated  that  external  services  are  justifiable  for  certain  departments  that  are  unique  in  their  scope,  function, 
and  needs  for  their  websites.  However,  DTIS  does  not  make  or  propose  policy  with  respect  to  who  should  or 
should  not  use  website  services  outside  of  DTIS 
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external  websites.  These  12  departments  estimate  that  they 
spend  approximately  $69,000^  annually  to  host  and  support 
their  external  websites.  DTIS  believes  that  this  is  low  and  has 
estimated  that  a  department  typically  spends  between  $12,000 
and  $18,000  a  year  to  design,  host,  and  maintain  an  external 
website.  We  estimated  what  these  12  departments  are  paying 
based  on  their  estimated  cost,  plus  a  labor  cost  that  includes 
0.5  full-time  equivalent  (FTE)  of  a  1052  Information  Systems 
(IS)  Business  Analyst  for  departments  that  support  their 
website  internally  and  0.1  FTE  of  a  1052  IS  Business 
Analyst  for  vendor-supported  websites.  The  table  below 
shows  what  these  departments  pay  DTIS  annually  for  web 
presence  and  our  estimates  of  what  it  likely  costs  them  to 
maintain  and  support  their  external  websites. 


I  Duplication  of  Web  Service  Cost 


Department 

Amount  Paid  to 
DTIS  (FY  2005-06) 

Estimated 
Additional  Cost  of 
External  Website 

Adult  and  Aging  Department  and  Commission 

Funded  through  MSA 

S  23,490 

Arts  Commission 

$  9,910 

9,937 

Children,  Youth  &  Their  Families  Department  and  Commission 

19,820 

48,057 

Environment  Department  (including  Urban  Forest  Council) 

9,910 

23,396 

Human  Services  Agency 

14,865 

23,490 

Municipal  Transportation  Agency  (including  Municipal  Railway) 

9,910 

11.396 

District  Attorney 

9,910 

9,936 

Public  Health  Department  and  Commission 

19,820 

86,979 

Public  Utilities  Commission 

9,910 

17.196 

Reproduction  and  Mail  Services 

14,865 

11,896 

Sheriff 

9,910 

9,996 

War  Memorial/Performing  Arts  Center 

9,910 

9,696 

Total 

$138,740 

$285,465 

Source:  DTIS  billing  records,  department's  estimates,  and  selected  labor  rates 


DTIS  and  Reproduction  Services  are  part  of  the  General 
Services  Agency,  and  both  entities  provide  website  services, 
although  maintenance  of  the  City's  website  is  a  specific 
responsibility  of  DTIS.  Clients  who  purchased  website 
services  from  Reproduction  Services  also  paid  DTIS  for 
service,  which  resulted  in  duplicate  payment  for  website 
services  that  are  already  available  to  clients  and  inefficient  use 
of  City  resources. 


^  $40,000  of  this  figure  was  the  Department  of  Public  Health's  estimate;  the  remaining  S29.000  was  for  the  1 1 
other  departments. 
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The  12  departments  consistently  cited  the  following  reasons 
for  not  using  DTIS  to  provide  all  of  their  website  services: 

•  They  want  control  over  their  site 

•  They  want  to  ensure  a  high  quality  of  customer  service 

•  They  do  not  like  the  DTIS  site  layout  and  its  capabilities 

However,  DTIS  staff  stated  that  their  content  management 
software  makes  it  very  simple  to  control  a  website  and  ensure 
a  high  quality  product  while  allowing  for  seamless  usability. 
Usability  is  important  because  it  provides  a  familiar  interface 
across  all  City  websites,  thereby  increasing  overall  efficiency 
and  the  likelihood  of  users  returning  to  the  site  and  knowing 
how  to  navigate  through  it.  DTIS  also  provides  a  level  of 
security  through  its  redundant  network,  and  departments 
outside  of  www.sfqov.orq  might  not  have  the  same  level  of 
protection  that  is  offered  by  an  enterprise-wide  system. 

Recommendations: 

9.  DTIS  should  consolidate  all  its  help  desk  functions  to 
provide  clients  a  single  point  of  contact  and  achieve  cost 
savings  and  efficiencies.  DTIS  should  standardize  its 
policies  and  procedures  for  its  help  desk  functions  and 
begin  to  aggregately  track  performance  measures  such  as: 

•  Number  of  requests 

•  Average  resolution  time  for  each  ticket  (by  priority) 

•  Percentage  of  requests  resolved  by  the  help  desk 
industry  standard  of  "First  Call  Resolution" 

DTIS  should  also  track  information  that  allows  it  to  make 
proactive  decisions  regarding  the  most  frequent  types  of 
requests.  Reports  or  queries  from  the  help  desk  software 
should  allow  DTIS  to  determine: 

•  The  most  common  (category  or  problem)  type  of 
requests 

•  The  most  frequent  location  of  requests 

•  The  most  frequent  caller  departments 

10.  DTIS  should  perform  a  staffing  and  capacity  analysis  to 
determine  how  it  can  overcome  the  current  obstacles  to 
integration  and  efficiency  and  consolidate  its  operator  and 
monitoring  functions.  DTIS  should  eliminate  those 
responsibilities  that  can  be  effectively  performed  by  others 
(e.g.,  after-hours  call  answering  that  will  be  better  handled 


by  operators  through  the  3-1-1  system)  and  focus  on  its 
key  strengths  and  areas  of  technical  expertise  like 
nnonitoring  and  troubleshooting. 

1 1 .  DTIS  should  work  with  the  Office  of  Contract 
Administration  and  other  departments  to  identify  all  fire  and 
burglar  alarm  monitoring  and  maintenance  being 
performed  by  vendors.  DTIS  should  identify  its  capacity  to 
monitor  and  maintain  additional  alarms,  estimate  the  cost 
savings  associated  with  acquiring  the  alarm  work,  pnoritize 
the  additional  alarm  work  that  it  has  the  capacity  to  pursue, 
develop  a  work  plan,  and  coordinate  with  departments  to 
acquire  responsibility  for  the  work.  If  the  cost  savings  that 
can  be  achieved  by  performing  the  work  in-house  exceed 
DTIS'  existing  capacity  to  perform  the  work,  DTIS  should 
prepare  and  submit  a  budget  allocation  to  the  Mayor's 
Office  with  justification  based  on  the  long-term  cost 
savings. 

12.  DTIS  should  draft  a  citywide  policy  regarding  the  use  of 
DTIS  versus  other  sources  for  website  services,  which 
should  include  criteria  for  when  departments  may  use  a 
website  outside  of  www. sf gov. org.  DTIS  should  seek 
formal  adoption  of  the  policy  from  the  Mayor's  Office 
and/or  COIT.  Once  adopted,  DTIS  should  stop  charging 
for  a  single  page  of  website  presence  when  departments 
meet  the  criteria.  DTIS  should  also  ensure  that  all  other 
departments  use  the  citywide  system  for  website  services, 
explore  ways  to  allow  the  departments  more  flexibility  in 
designing  and  updating  their  websites,  and  work  with 
departments  to  address  their  concerns  regarding  website 
development  and  content  so  they  can  discontinue  their  use 
of  external  support. 

13.  The  General  Services  Agency  should  consolidate  the 
website  services  it  provides  to  departments  so  that  one 
single  group  within  the  agency  provides  the  sen/ices  and 
to  ensure  that  clients  are  not  paying  both  DTIS  and 
Reproduction  Services. 

Finding  1.4.  No  Standard  Project  Management 
Approach  or  Current  Project  Listings  Exist 

DTIS  could  not  ensure  that  it  was  appropriately  managing  its 
portfolio  of  projects  because  it  did  not  use  standard  project 
management  tools  or  maintain  a  current  inventory  of  its 
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application  development  and  DBA  projects.  The  lack  of 
standard  and  consistent  project  management  tools  may  lead 
to  inefficient  use  of  resources,  ineffective  outcomes,  or  even 
failure  for  many  of  its  $1 8  million  worth  of  application 
development  and  DBA  projects  that  it  conducts  each  year. 

A  current  list  of  projects  would  provide  a  formal  mechanism  for 
staff  to  be  aware  of  what  others  are  working  on,  which  would 
allow  staff  to  collaborate  on  projects  that  have  similar 
requirements  or  technical  components.  It  would  also  allow 
DTIS  to  seek  opportunities  to  develop  citywide  application 
solutions  by  identifying  similar  requirements  or  technical 
components  of  projects.  While  this  is  still  possible  without  a 
written  list  of  projects,  a  list  makes  collaboration  more  .iKely 
because  it  provides  a  reference  point  for  all.  A  project 
inventory  is  a  basic  management  tool  and  some  cities  and 
counties  post  them  on  their  websites  to  give  clients  and  the 
public  a  sense  of  the  type  and  scope  of  work  being  conducted. 

DTIS  spent  $102,400  to  train  its  project  managers  and  develop 
a  project  management  office,  and  also  developed  an  internal 
project  management  manual  for  its  application  development 
and  DBA  services.  However,  DTIS  discontinued  funding  for 
the  office  before  it  was  implemented  and  project  managers  do 
not  use  either  the  established  methodology  or  a  generally 
recognized  project  management  approach. 

A  recent  study  by  the  Center  for  Project  Management  showed 
that  robust  project  management  techniques  are  imperative  to 
avoid  costly  mistakes  and  a  corresponding  waste  of  resources. 
The  study  showed  that  nearly  40  percent  of  IT  projects  are 
abandoned  before  completion  and  that  80  percent  of 
managers  do  not  identify  criteria  to  critically  evaluate  their 
projects.  Two  common  project  management  tools  are  service 
level  management  and  service  level  agreements  (SLAs). 
Service  level  management  is  a  technique  used  to  measure  IT 
processes  in  a  consistent  and  verifiable  manner  and  often  use 
SLAs  as  an  enforcement  instrument.  SLAs  formalize  the 
expectations  of  a  service  by  defining  the  roles  and 
responsibilities  for  each  party  to  mitigate  the  possibility  of 
misunderstandings  about  a  project's  scope  and  expected 
outcome. 

DTIS'  project  managers  use  varying  methods  to  manage  their 
projects  and  budgets,  which  include  working  on  a  close  but 
informal  basis  with  client  departments  to  assess  the  needs  of 


each  project.  The  project  managers  rely  on  oral 
communications  with  clients  and  some  use  written  project 
plans  -  a  list  of  tasks  and  schedules.  As  a  result,  DTIS  cannot 
ensure  that  it  has  mutual  agreement  with  its  clients  regarding 
the  expected  outcomes  of  each  project,  that  it  is  spending  its 
resources  to  achieve  the  results  expected  by  its  clients,  or  that 
it  can  evaluate  the  success  or  failure  of  each  project. 

Recommendations: 

14.  DTIS  should  implement  a  generally  recognized  project 
management  approach  that  includes  performance 
measurement,  service  level  agreements,  and  other  project 
management  tools  that  ensure  accountability. 

15.  DTIS  should  develop  and  maintain  an  inventory  listing  and 
description  of  its  application  development  and  DBA 
projects.  The  listing  should  be  available  to  staff  to  raise 
awareness  of  what  others  are  working  on  and  encourage 
collaboration  and  discussion  of  solutions  for  individual 
projects  and  citywide  applications. 
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CHAPTER  2  -  PERFORMANCE  MEASURES 


Finding  2.1.  Performance  Measures  Not  Meaningful 
and  Reported  Results  Not  Reliable 

We  reviewed  DTIS'  19^  performance  measures  and  rated 
each  one  based  on  its  meaningfulness  and  the  reliability  of  its 
reported  results.  DTIS  reported  reliable  and  accurate  results 
for  only  three  measures  and  only  ten  measures  were 
meaningful  and  useful.  Without  reliable  and  meaningful 
performance  measurement  information,  it  is  difficult  for  DTIS  to 
identify  its  successes  and  areas  needing  improvement.  DTIS 
cannot  use  performance  information,  or  will  use  erroneous 
information,  to  set  goals,  allocate  resources,  help  managers 
make  decisions,  and  improve  delivery  of  services. 

Reasons  for  the  reported  results  not  being  reliable  or  accurate 
varied: 

•  What  the  measure  is  supposed  to  gauge  varied  from  what 
the  reported  result  actually  represented.  For  example, 
DTIS  reported  that  99  percent  of  its  website  visitors  found 
timely  and  valuable  information.  However,  DTIS  counted 
each  website  visitor  who  did  not  send  an  e-mail  complaint 
to  the  webmaster  as  receiving  timely  and  valuable 
information  and  calculated  the  measure  based  on  the 
number  of  complaints  received  rather  than  on  positive 
assertions  from  website  visitors. 

•  DTIS  used  estimates  rather  than  calculations  to  report 
some  results.  For  example,  DTIS  estimated  that 

80  percent  of  repairs  for  portable  radios  were  completed 
within  the  same  day  but  did  not  calculate  the  actual 
percentage. 

•  DTIS  did  not  track  the  information  necessary  to  calculate 
some  results.  For  example,  DTIS  reported  that  it  prepared 
and  delivered  100  percent  of  its  telecom  policy  studies  or 
recommendations  according  to  agreed  upon  target  dates. 
However,  DTIS  did  not  track  the  dates  of  completion  or  the 
agreed  upon  dates. 

•  DTIS  reported  results  based  on  disparate  or  incomplete 
information.  For  example,  DTIS  reported  that  98  percent 
of  service  requests  were  completed  within  the  budget 


*  We  did  not  review  the  meaningfulness  of  two  measures  and  the  reliability  of  four  measures  because  DTIS  does 
not  have  a  mechanism  to  measure  or  could  not  provide  information  to  rate  the  measures. 
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agreed  upon  with  the  client.  However,  the  result  reported 
was  based  on  incomplete  information  and  each  project 
manager  calculated  "completed  within  budget"  using 
different  data.  Some  based  it  on  the  budget  of  the  specific 
project  code  and  others  based  it  on  what  they  told  the 
client. 

Reasons  for  certain  measures  not  being  meaningful  or  useful 
also  varied: 

•  DTIS  used  an  inappropriate  measure,  which  hid  significant 
differences  in  performance.  For  example,  DTIS  measured 
reliability  of  systems  as  a  percentage  of  "uptime"  during  the 
year  rather  than  the  actual  time  a  system  was  down. 
Reliability  reported  at  99  percent  could  mean  that  a  system 
was  down  any  amount  of  time  between  31.5  seconds  and 
87.6  hours  (3.65  days). 

•  DTIS  lacked  influence  over  the  result.  For  example,  DTIS 
reported  that  100  percent  of  all  cable  franchise  complaints 
from  subscribers  were  satisfactorily  resolved  within  24 
hours.  However,  DTIS  does  not  resolve  the  issue  or 
advocate  for  clients.  DTIS  counts  every  call  as 
satisfactorily  resolved  even  though  it  does  not  do  anything 
with  the  compliant,  not  even  refer  the  call  to  the  cable 
franchise. 

The  causes  of  deficiencies  in  reliability  and  meaningfulness 
include: 

•  DTIS  has  not  assumed  ownership  of  its  performance 
measures  and  results.  DTIS  staff  and  management 
indicated  that  they  had  inherited  measures  from  their 
predecessors.  Some  staff  recognize  the  deficiencies  with 
the  measures  and  the  need  for  improvement  but  have  not 
proposed  new  measures  or  changed  the  current  measures 
so  that  they  gauge  more  meaningful  outcomes. 

•  As  DTIS'  systems,  processes,  and  functions  have  changed 
over  the  years,  corresponding  performance  measures 
have  not  been  developed  and  the  methods  and  data 
sources  required  to  calculate  the  existing  performance 
measures  have  not  been  updated  or  maintained. 

•  Various  people  within  the  department  calculate  the 
measures,  but  DTIS  lacks  a  review  process  to  assess  the 
reliability  of  the  results  reported  or  the  usefulness  of  the 
measures. 


The  City's  Administrative  Code  requires  departments  to 
establish  performance  indicators  to  measure  or  assess  the 
relevant  outputs,  service  levels,  and  outcomes  of  each 
program  activity.  DTIS  reports  its  performance  measures  mid- 
year and  annually  to  the  Controller's  Office  and  reports  1 1  of 
its  measures  each  year  in  the  Mayor's  Proposed  Budget. 
DTIS  defines  performance  management  as  the  use  of 
performance  information  to  set  goals,  allocate  resources,  help 
managers  make  decisions,  and  report  results;  it  is  intended  to 
help  DTIS  improve  delivery  of  services.  DTIS  states  that  it 
links  its  strategic  priorities  to  performance  goals  and  is  working 
towards  developing  measures  on  all  levels. 

DTIS  may  make  inappropriate  management  decisions 
because  the  lack  of  reliable  and  accurate  performance  results 
may  not  reveal  areas  that  need  improvement  or  may 
erroneously  indicate  that  an  area  is  performing  well.  For 
example,  resources  may  not  be  allocated  to  improving  DTIS' 
systems  because  they  are  assumed  to  be  sufficiently  reliable 
based  on  the  reported  percentage  of  uptime.  However,  since 
a  system  could  be  down  for  over  80  hours  a  year  and  be 
reported  as  reliable  99  percent  of  the  time,  DTIS  and  other 
City  decision-makers  may  have  a  false  level  of  confidence 
about  the  actual  reliability  of  a  system.  The  City  cannot  afford 
critical  systems,  such  as  those  for  public  safety,  to  be  down  for 
hours,  even  if  the  reliability  is  99  percent. 

The  lack  of  meaningful  performance  measures  may  also 
cause  DTIS  to  miss  opportunities  to  improve  its  performance 
and  market  its  services  to  departments.  If  DTIS  can  reliably 
measure  the  percentage  of  service  requests  that  it  completes 
within  budget  or  show  that  it  can  perform  at  a  lower  cost  than  a 
vendor,  it  could  use  this  information  to  improve  its  adherence 
to  budget  and  to  market  its  service  to  departments.  For 
example,  DTIS  provided  information  indicating  that  it  had 
performed  wireless  service  projects  for  much  less  than  what 
vendors  would  have  charged  the  City,  but  this  type  of  success 
is  not  captured  in  any  of  DTIS'  performance  measures. 

Recommendations: 

16.  First,  DTIS  should  delete  all  measures  rated  as  not 
meaningful.  These  measures  consume  time  and 
resources  to  track,  but  the  results  do  not  reflect  the 
department's  performance  and  the  information  is  of  little  or 
no  use  to  the  department. 
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17.  Second,  DTIS  should  review  the  remaining  measures  and 
determine  if  the  reported  results  accurately  reflect 
performance.  Measures  where  significant  differences  in 
performance  do  not  properly  reflect  the  result  should  be 
either  be  deleted  or  changed  so  the  results  reflect  actual 
performance. 

18.  DTIS  should  identify  its  key  functions  and  operations  and 
ensure  that  it  has  meaningful  measures  for  each  as 
required  by  the  Administrative  Code.  DTIS  should  ensure 
that  any  new  measures  are  meaningful  to  the  department 
and  accurately  reflect  the  department's  functions. 

19.  After  performance  measures  have  been  selected  and 
edited,  DTIS  should  evaluate  its  systems  to  ensure  that  it 
can  measure  and  report  accurate  (i.e.,  +/-  5%)  results. 
DTIS  should  document  its  procedures  for  calculating  the 
results  for  each  measure.  DTIS  should  retain 
documentation  that  supports  the  reported  results  for  a 
minimum  of  three  years  and  use  the  results  internally  to 
monitor  and  make  improvements  in  its  performance  over 
time. 

20.  DTIS  should  establish  a  process  to  review  the  continued 
accuracy  and  meaningfulness  of  its  performance  measures 
on  at  least  a  yearly  basis.  DTIS  should  delete  or  revise 
measures  that  are  no  longer  meaningful  and  update  its 
procedures  for  collecting  information  and  measuring 
performance  to  ensure  the  continued  reliability  of  results. 

21 .  DTIS  should  explicitly  obtain  management  and  staff  buy-in 
for  accurate,  reliable,  and  meaningful  performance 
measurement.  DTIS  should  take  firm  ownership  of  its 
performance  measures  and  should  establish  accountability 
and  responsibility  for  the  results  of  its  measures  among  the 
various  groups  within  DTIS. 


CHAPTER  3  -  BUILDING  SECURITY 


Finding  3.1.  Restricted  Areas  at  901  Rankin  Easily 
Accessible 

DTIS  has  after-hours  security  for  its  facility  at  901  Rankin,  but 
restricted  areas  were  open  and  unattended  duhng  business 
hours.  We  walked  into  restricted  areas,  photographed 
equipment,  and  observed  that  it  would  have  been  easy  to  take 
equipment  and  not  be  noticed.  These  security  weaknesses 
existed  despite  approximately  $43,000  worth  of  losses  at 
901  Rankin  in  the  last  three  years  due  to  break-ins  and 
vandalism.  Losses  included  parking  meters,  wiring  cable,  and 
extensive  damage  to  service  vehicles.  DTIS  and  the 
Department  of  Parking  and  Traffic  currently  share  this  facility, 
which  is  used  to  store  City  equipment  and  supplies.  The 
following  pictures  show  unsecured  exterior  and  interior  areas 
that  we  accessed. 


Source:  Audit  staff  photos 

The  site  is  currently  equipped  with  cameras  that  are  monitored 
by  DTIS  staff  at  1011  Turk.  DTIS  installed  motion  detection 
devices  on  these  cameras  on  a  pilot  basis  to  assist  them  in 
detecting  intruders.  DTIS  also  hired  a  security  guard  company 
to  monitor  the  site  after  hours.  The  security  guards  have 
radios  to  communicate  with  the  staff  at  1011  Turk  who  monitor 
the  cameras.  Despite  this  investment  in  security,  DTIS  staff 
do  not  monitor  the  cameras  duhng  business  hours  and 
security  guards  only  patrol  the  site  after  hours. 

Any  theft  that  may  occur  dunng  business  hours  could  go 
unnoticed  because  DTIS  does  not  have  an  inventory 


Unsecured  Areas 
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management  system  and  does  not  take  inventory  of  its  on-site 
equipment  and  supplies.  Without  an  inventory  management 
system,  DTIS  cannot  determine  if  items  have  been  lost,  stolen, 
or  missing  and  might  not  be  able  to  calculate  the  dollar  loss 
associated  with  missing  equipment  and  supplies. 

Recommendations: 

22.  DTIS  should  coordinate  with  the  Department  of  Parking 
and  Traffic  to  implement  procedures  to  strengthen  security 
at  901  Rankin  and  to  ensure  that  resthcted  areas  are 
monitored  and  that  access  to  these  areas  is  limited  during 
all  hours. 

23.  DTIS  should  develop  an  inventory  management  system  to 
track  the  quantity  and  value  of  equipment  and  supplies  at 
its  facilities. 


Appendix  A  -  Performance  Measures 


Methodology:  We  reviewed  the  FY  2005-06  year-end  results  for  the  19  performance 
measures  that  DTIS  will  report  in  the  Controller's  Office  performance  measure  database. 
Eleven  of  these  measures  are  typically  reported  in  the  Mayor's  Proposed  Budget.  We 
reviewed  the  calculation  methods  and  data  sources  that  DTIS  used  to  report  results  for  each 
performance  measure  and  evaluated  each  measure  to  determine  the  reliability/accuracy  of 
the  reported  results  and  the  meaningfulness/usefulness  of  each  measure.  We  based  our 
scoring  system  on  the  certification  techniques  developed  by  Maricopa  County,  AZ,  but  used 
two  separate  ratings  because  we  considered  reliability/accuracy  and  meaningfulness/ 
usefulness  as  different  indicators  of  the  quality  of  a  performance  measure.  For  each 
measure,  we  evaluated  the  complexity,  consistency  of  definition  and  data  source,  internal 
controls,  use  of  automation,  usefulness  to  the  department,  the  amount  of  influence  DTIS  has 
over  the  result,  and  whether  it  measures  a  service  or  outcome  of  the  department.  Based  on 
the  scores,  we  assigned  each  measure  to  a  rating  category.  The  categories  are: 


Reliability  and  Accuracy  of  Reported  Results 

Category 

Definition 

Premium 
Rating 

Result  is  accurate  (+/-  5%)  or 
reliable,  and  adequate  procedures 
are  in  place  to  ensure  reliability. 

"With 

Qualifications" 
Rating 

High  level  of  confidence  about  the 
accuracy  of  the  result  with  some 
minor  reservations  about  the  data; 
procedures  to  ensure  reliability  and 
accuracy  of  the  result  can  be 
improved. 

Factors 
Prevented 
Rating  as 
Reliable  or 
Accurate 

Could  not  rate  the  measure  as 
reliable  due  to  inadequate 
procedures,  insufficient 
documentation,  or  multiple 
reservations  and  qualifications  about 
the  information  or  processes.  We 
might  have  been  able  to  rate  the 
measure  as  reliable  if  additional 
information  had  been  available  or  if 
we  could  have  validated  certain 
pieces  of  information  used  to 
calculate  the  result. 

Unreliable 

and/or 

Inaccurate 

Performance  measure  result  is  not 
accurate  (+/-  5%)  or  is  unreliable 
because  of  significant  reservations 
and  qualifications. 

IVIeaningfulness  and  Usefulness  of  Measures 

Category 

Definition 

Premium 
Rating 

Measure  is  a  good  indicator  of  performance 
and  gauges  the  actual  services  and  outcomes 

of  the  department. 

"With 

Qualifications" 
Rating 

Measure  is  a  good  indicator  of  performance 
with  minor  exceptions  in  some  of  the  following: 

•  Department  uses  result  to  evaluate  specific 
functions 

•  Measures  actual  services  and  outcomes 

•  Result  is  influenced  directly  by  the 
department's  efforts  and  actions 

•  Measure  can  be  easily  understood  in  terms 
of  what  it  is  actually  measuring 

Factors 
Prevented 
Rating  as 
Meaningful  or 
Useful 

Several  qualifications  and  reservations 
regarding: 

•  Department  uses  result  to  evaluate  specific 

functions 

•  Measures  actual  services  and  outcomes 

•  Result  is  influenced  directly  by  the 
department's  efforts  and  actions 

•  Measure  can  be  easily  understood  in  terms 
of  what  it  is  actually  measuring 

We  might  have  been  able  to  rate  the  measure 
as  meaningful  if  additional  information  had 
been  available. 

Not  Meaningful 
or  Useful 

Little  to  no  use:  recommend  eliminating 

A-1 


USEFULNESS  OF 
ES 

Not 
Meaningful 
and/or  Useful 
(<40%) 

DTIS  HAS  NO  MECHANISM  TO  MEASURE  RESULTS 

DTIS  HAS  NO  MECHANISM  TO  MEASURE  RESULTS 

Factors 
Prevented 
Rating  as 
Meaningful 
(69%-40%) 

MEANINGFULNESS  AND 
MEASURI 

"With 
Qualifications" 
Rating 
(89%-70%) 

Premium 
Rating 
(100%-90%) 

Raw 
Score 

100% 

o 

o 

o 

o 

RELIABILITY  AND  ACCURACY  OF  REPORTED 
RESULTS 

Inaccurate 
and/or 

Unreliable 
(<40%) 

Could  not  evaluate  because  DTIS  did  not  provide 
sufficient  information. 

Could  not  evaluate  because  DTIS  did  not  provide 
sufficient  information. 

Factors 
Prevented 
Rating  as 

Reliable 
(69%-40%) 

"With 
Qualifications" 
Rating 
(89%-70%) 

Premium 
Rating 
(100%-90%) 

Raw 
Score 

o 

o 

V/N 

V/N 

%0 

o 

Performance  Measure 

Average  number  of  days  from  the  end  of  the 
prior  month  to  complete  interdepartmental 
project  billing 

Average  number  of  days  to  respond  to  client 
requests  for  information  on  telephone  billing 

Reliability  of  the  PBX  network  managed  by 
DTIS 

Reliability  of  Data  Center  backbone 

Percentage  of  customer  complaints 
resolved  within  8  hours  of  receipt  from 
customer 

Percentage  of  trouble  tickets  resolved  by 
Help  Desk  -  Industry  standard  of  "First  Call  | 
Resolution" 

Percentage  of  services  that  have  internal 
escalation  procedures  in  place 

Percentage  of  backups  successfully 

restored 

Average  customer  rating  of  Hot  Site 

effectiveness 

Percentage  of  telecom  and  technology 
related  policy  decision  material  research 
studies  or  recommendations  prepared  and 
delivered  according  to  agreed  upon  target 
dates 

Percentage  of  all  franchise  complaints  by 
subscribers  with  a  satisfactory  resolution 
within  24  hours 

Percentage  of  service  requests  completed 
by  the  date  agreed  upon  with  the  client 

(D 

o 

MEANINGFULNESS  AND  USEFULNESS  OF 
MEASURES 

Not 
Meaningful 
and/or  Useful 
(<40%) 

Factors 
Prevented 
Rating  as 
Meaningful 
(69%-40%) 

"With 
Qualifications" 
Rating 
(89%-70%) 

Premium 
Rating 
(100%-90%) 

en 

O) 

Raw 
Score 

h- 

o 

CT) 

in 

o 

O 

00 

1  N/A 

N/A 

RELIABILITY  AND  ACCURACY  OF  REPORTED  1 
RESULTS  1 

Inaccurate 
and/or 

Unreliable 
(<40%) 

V- 

CM 

o 

00 

Factors 
Prevented 
Rating  as 

Reliable 
(69%-40%) 

O 

o 

"With 
Qualifications" 
Rating 
(89%-70%) 

Premium 
Rating 
(100%-90%) 

CM 

Raw 
Score 

o 

CO 

cn 

00 

o 

o 

V/N 

Performance  Measure 

Percentage  of  service  requests  completed 
within  the  budget  agreed  upon  with  the 
client 

Percentage  of  the  regular  Board  of 
Supervisors'  meetings  carried 

Availability  of  24-hour  government 
informational  programming  on  Cable 
Channel  26 

Percentage  of  web  site  visitors  who 
reported  that  they  found  timely  and  valuable 
information 

Reliability  of  the  public  safety  radio  and 
wireless  data  system 

Percentage  of  repairs  for  portable  and 
mobile  radios  completed  within  the  same 
day  of  the  request 

Percentage  of  repairs  for  mobile  data 
terminals  completed  within  the  same  day  of 
the  request 

TOTALS 

PERCENTAGE  OF  TOTAL  MEASURES 
EVALUATED 

CO 

•"4- 

in 

CD 

CO 

APPENDIX  B  -  DEPARTMENT  RESPONSE  TO  AUDIT 


City  AND  County  of  San  Francisco 


Department  of  Telecommunications 
AND  Information  Services 


Chris  Vein 

Chief  information  Officer 


Tdephone:  (413)554-0801 


Chief  Administrative  Officer 


Telephone:  (415)  554-0803    Fax:  (415)  554^733 


November  13.  2006 


Pablo  Federic» 
Performance  Audit  Manager 
Office  of  the  Controlier 
1  Dr.  Carlton  B.  Goodlett  Place 
Room  388 

San  Francisco,  CA  94102 
Dear  Mr.  Federico: 

Please  find  attached  the  Audit  Recommendation  and  Response  Form  relating  to  the  Office  of 
the  Controller,  City  Services  Audit  of  the  Department  of  Telecommunications  and  Information 
Services  (DTIS). 

DTiS  has  relentlessly  focused  on  improving  our  perfomiance.  As  you  recognized  in  the  report, 
DTIS  recently  initiated  a  major  restructuring  of  our  organization,  staffing,  and  processes  and 
procedures.  That  restructuring  will  address  all  of  your  recommendations  as  well  as  other  areas 
not  covered  by  your  audit. 

Specifically,  we  have  developed  a  three-year  plan  to  radically  change  the  funding  model  of  DTIS 
and  the  rate  model  used  to  recover  costs.  Recognizing  that  DTIS  must  ensure  and  track  a  high 
level  of  customer  service,  a  new  division  has  been  created  that  focuses  solely  on  our  pursuit  of 
improvement.  Finally,  DTIS  takes  the  security  of  City  assets  and  employees  very  seriously  and 
are  in  the  process  of  improving  even  more  our  Rankin  Street  facility. 

It  has  been  a  pleasure  working  with  you  and  your  team  over  the  last  6  months.  We  look 
forward  to  future^cussions  on  DTfS  perfonnance  and  the  improvements  we  are  making. 


CfVfSA.  Vein 
Chief  Infonnation  Officer 


cc:  Edwin  M.  Lee,  City  Administrator 


875  Stevenson  street.  Fifth  Floor  •  San  Francisco,  CA  94103-0948 
Office:  (415)554-0800  •  Facsimile:  (415)554-4733 
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